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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  will  be  a  “paper  only”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resolution,  with  the  attendant  restrictions  on  travel  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
colleagues  and  the  opportunity  for  all  our  researchers  to  present  their  work.  However,  we 
intend  to  simulate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  published  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduled  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Naval  Postgraduate  School  (NPS)  continues  at  a  normal  pace.  Since  the  ARP’s 
founding  in  2003,  over  1,200  original  research  reports  have  been  added  to  the  acquisition 
body  of  knowledge.  We  continue  to  add  to  that  library,  located  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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Abstract 

The  current  Enterprise  Requirements  and  Acquisition  Model,  a  discrete  event  simulation  of 
the  major  tasks  and  decisions  within  the  DoD  acquisition  system,  identifies  several  what-if 
intervention  strategies  to  improve  program  completion  time.  However,  processes  that 
contribute  to  the  program  acquisition  completion  time  were  not  explicitly  identified.  This 
research  seeks  to  determine  the  acquisition  processes  that  contribute  significantly  to  the  time 
a  program  reaches  Milestone  (MS)  B  and  provide  interventions  to  improve  program 
completion  time.  In  order  to  solve  this  problem,  this  research  uses  critical  path  analysis  to 
determine  the  bottleneck  activities  in  the  Pre-MS  B  processes  using  additional  simulation 
analysis.  Results  show  that  the  systems  engineering  processes  are  the  bottleneck  activities 
in  Pre-MS  B  acquisition  stage.  Furthermore,  this  research  then  examines  the  effect  of  these 
processes  by  varying  the  mean  completion  times  and  having  them  occur  earlier  in  the 
acquisition  process.  Potential  policies  are  formulated  from  the  results  to  further  reduce 
program  acquisition  completion  time. 
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Introduction 

A  large  number  of  Department  of  Defense  (DoD)  projects  are  being  completed 
behind  schedule  and  over-budget  (Schwartz,  2010).  To  support  this  claim,  a  Government 
Accountability  Office  (GAO)  report  released  in  2009  stated  that  for  the  DoD’s  2008  portfolio, 
on  average  a  program  faced  a  22-month  delay  and  exceeded  the  original  budget  (Sullivan  et 
al.,  2009).  Generally,  total  cost  growth  of  44%  has  been  consistent  over  the  past  few 
decades  with  a  recent  assessment  by  RAND  (Arena  et  al.,  2006).  Hence,  potential 
intervention  strategies  and  policies  to  improve  the  acquisition  processes  would  be 
worthwhile.  On  the  other  hand,  since  the  end-to-end  DoD  acquisition  process  is  a  large, 
complex,  socio-technological  system,  it  is  difficult  to  analyze  and  determine  which  processes 
or  factors  affect  performance  metrics  like  time,  cost,  and  resource  utilization.  The  current 
DoD  acquisition  system,  which  is  composed  of  three  separate  and  distinct  processes — the 
Joint  Capabilities  Integration  Development  System  (JCIDS),  the  Planning,  Programming, 
Budgeting  &  Execution  (PPBE)  process,  and  the  formal  acquisition  development  system 
outlined  by  the  DoD  5000  series  of  instructions — does  not  exist  in  a  static  environment.  The 
system  is  constantly  being  adjusted,  either  through  policy  changes  or  statute  (CJCS,  2012; 
Weapon  System  Acquisition  Reform  Act  of  2009;  OUSD[AT&L],  2008).  Hence,  other  viable 
analysis  methodologies  must  be  utilized  to  fully  comprehend  this  complex  system. 

In  2009,  a  discrete  event  simulation  (DES)  model  called  the  Enterprise  Requirements 
and  Acquisition  Model  (ERAM)  was  created  by  Wirthlin  (2009).  This  model  was  created  to 
simulate  the  actual  acquisition  processes  of  the  DoD,  using  the  Air  Force  implementation  of 
acquisition  processes  as  the  basis  of  the  model,  in  order  to  provide  further  insight  and 
understanding  of  the  complex  system’s  behavior.  Furthermore,  ERAM  has  benefited  from 
additional  research  since  the  original  2009  Wirthlin  version  (Leach  &  Searle,  2011; 
Montgomery,  2012).  These  new  versions  have  added  additional  functionality  and  options  for 
model  users  to  manipulate  (Wirthlin,  Houston,  &  Madachy,  2011).  According  to  the  ERAM 
model,  during  the  acquisition  process,  approximately  80%  of  the  time,  a  program  was 
undergoing  parallel  processes  when  it  is  in  the  acquisition  system.  It  was  also  observed  that 
one  of  the  main  portions  of  the  model  during  which  these  parallel  processes  take  place  are 
within  the  Pre-Milestone  B  (Pre-MS  B)  stage.  However,  Wirthlin’s  research  did  not  identify 
the  significant  processes  that  affect  the  total  program  time  for  a  project  to  reach  MS  B. 

Against  this  background,  this  research  addressed  these  limitations  and  issues  by 
additional  simulation  and  statistical  analysis  on  the  ERAM  Arena  version  of  the  model.  The 
end  goal  of  this  research  was  to  determine  the  bottleneck  of  the  Pre-MS  B  processes, 
investigate  interventions  to  alleviate  the  bottleneck,  and  translate  them  into  implementable 
policy  changes.  The  rest  of  this  paper  is  organized  as  follows.  The  Review  of  Literature 
section  provides  an  overview  of  the  current  literature  on  bottleneck  analysis  and  the  ERAM 
model.  The  Simulation  Analysis  Methodology  section  presents  the  simulation  analysis 
methodology  performed,  while  the  Results  and  Discussions  section  shows  the  results  of  the 
analysis.  Finally,  the  Conclusions  section  presents  the  conclusions  of  this  research  as  well 
as  viable  intervention  policies  for  reducing  the  time  a  program  takes  to  reach  MS  B. 

Review  of  Literature 

The  ERAM  Model 

The  ERAM  simulation  model  extends  from  the  generation  of  capability  requirements 
in  the  JCIDS  process  to  MS  C,  the  review  before  the  production  stage  begins.  Additionally, 
the  ERAM  is  abstracted  at  a  very  high  level  (Wirthlin,  2009).  This  high  level  of  abstraction 
allows  overall  system  performance  to  be  more  easily  studied.  For  each  replication,  ERAM 
produces  schedule  time  for  programs  that  reach  MS  C.  Although  cost  is  not  measured,  it 
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was  found  that  cost  overruns  were  closely  related  to  schedule  overruns  (Wirthlin,  2009).  The 
validation  and  verification  of  ERAM  included  hand  modeling,  iterations  of  correction  from 
feedback  of  experts  in  all  three  systems  that  comprise  the  entire  acquisition  system,  and 
comparison  of  schedule  and  budget  information  from  the  DAMIR  and  SMART  databases  to 
distributions  of  the  schedule  time  of  model-generated  data  (Wirthlin,  2009). 

The  original  version  of  ERAM  was  created  in  Arena  Simulation  software;  however,  it 
was  translated  into  an  ExtendSim  version  (ERAM  1 .0)  to  serve  as  a  schedule  and  success 
estimation  tool  of  space  programs  for  the  Concept  Design  Center  of  Aerospace  Corporation 
(Leach  &  Searle,  2011).  Leach  and  Searle  further  modified  the  model  introducing  ERAM  1.1 
to  2.1  by  correcting  discrepancies  between  the  Arena  and  ExtendSim  models,  adding  user- 
controlled  variables,  incorporating  space-acquisition  specific  elements,  and  updating  the 
model  to  include  policy  in  the  newly  released  DoDI  5000.02  document.  Montgomery  (2012) 
continued  developing  the  model  in  order  to  add  the  rapid  acquisition  process  and  include 
ACAT  ll/lll  programs.  A  summary  of  the  versions  of  the  ERAM  is  presented  in  Table  1 . 


Table  1.  ERAM  Versions  Adapted  From  Houston  (2012) 


Author 

Version 

Number 

Changes 

Wirthlin  (2009) 

ERAM  1.0 

Baseline  Translation  from  Arena  to  ExtendSim 

ERAM  1.1 

Updates  by  the  Aerospace  Design  Team  and 

Served  as  new  baseline  model 

Leach  and  Searle 

ERAM  1.2 

Implemented  new  DoD  5000.02  policies 

(2011) 

ERAM  2.0 

Incorporated  the  global  variables  that  modify 
acquisition  capabilities 

ERAM  2.1 

Incorporated  the  JCIDS  review  process 

Montgomery  (2012) 

ERAM  2.2 

Added  more  capabilities  for  ACAT  2/3  and  Rapid 
Acquisition  Process 

Since  the  ExtendSim  version  of  ERAM  was  designed  with  the  purpose  of  allowing 
Aerospace  Corporation  to  create  estimates  of  the  schedule  and  success  of  a  particular 
project,  it  has  a  distinctly  different  scope  and  utility  from  the  Arena  model  of  ERAM.  The 
Arena  model  allows  the  user  to  view  the  behavior  of  the  overall  portfolio  while  the 
ExtendSim  version  allows  the  user  to  investigate  a  specific  program.  For  example,  while  the 
ExtendSim  requires  the  user  to  select  a  specific  ACAT  level  for  the  program  being  tested, 
the  Arena  version  assigns  ACAT  levels  based  on  the  distribution  of  programs  observed  in 
the  actual  acquisition  system.  While  the  ExtendSim  version  of  ERAM  was  designed  with  the 
intention  of  allowing  the  user  to  perform  what-if  scenarios,  as  far  as  the  researcher  is 
concerned,  no  literature  of  the  evaluation  of  possible  intervention  strategies  using  the 
ExtendSim  version  of  ERAM  has  been  published.  In  his  dissertation,  Wirthlin  investigated 
the  effect  of  20  interventions  on  the  effect  of  end-to-end  acquisition  time  in  the  Arena 
version.  When  all  20  interventions  were  implemented,  a  20%  reduction  in  end-to-end 
acquisition  time  was  achieved.  However,  more  interventions  can  be  developed  to  further 
study  and  improve  the  DoD  end-to-end  acquisition  process. 

Critical  Path  Analysis 

To  the  best  of  our  knowledge,  no  literature  has  attempted  to  identify  the  critical  path 
of  the  acquisition  process  (Monaco  &  White,  2005).  Although  long  cycle  times  continue  to 
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plague  DoD  acquisition  programs,  relatively  few  studies  have  focused  on  identifying 
significant  processes  that  dictate  program  cycle  time.  Despite  the  Packard  Commission’s 
assertion  that  schedule  drives  costs,  most  studies  and  policy  changes  have  focused  on  cost 
reduction  rather  than  reducing  cycle  time  (Al-Harbi,  2001;  McNutt,  1999).  Drezner  and  Smith 
(1990)  performed  a  statistical  analysis  of  10  programs  in  order  to  hypothesize  factors  that 
affect  the  original  plan  and  program  deviation.  A  study  performed  by  Tyson,  Nelson,  Om, 
and  Palmer  (1989)  examined  schedule  variance  and  its  causes.  The  study  found  that 
prototyping,  sole-source  procurement,  fixed-priced  contracts,  and  multiyear  procurement 
reduced  schedule  variance.  The  study  also  found  that  programs  awarded  through  full  and 
open  competition  experience  more  schedule  growth  than  those  programs  that  did  not. 
Another  possible  schedule  driver  is  presented  by  Brown,  Flowe,  and  Hamel  (2007).  Brown 
et  al.  compared  the  schedule  quality  of  joint  and  single-system  programs.  From  this  study  it 
was  found  that  joint  system  programs  have  significantly  more  schedule  breaches;  however, 
the  research  did  not  identify  the  root  cause  of  this  difference  (Brown  et  al.,  2007). 

In  summary,  to  the  best  of  our  knowledge,  there  exists  no  research  that  has  been 
conducted  that  isolates  and  identifies  bottleneck  activities  and  its  effect  on  the  program 
completion  time  throughout  the  DoD  acquisition  process.  Hence,  intervention  strategies  to 
be  developed  must  be  focused  on  addressing  bottleneck  issues  to  obtain  maximum 
improvement  of  the  end-to-end  DoD  acquisition  process. 

Simulation  Analysis  Methodology 

This  section  describes  the  analysis  performed  to  identify  bottleneck  operations  within 
the  Pre-MS  B  stage.  After  identifying  bottleneck  operations,  intervention  strategies  were  also 
formulated  in  this  section  to  reduce  total  program  completion  time.  Hence,  this  research  was 
performed  in  two  phases.  A  brief  description  of  these  phases  is  presented  as  follows: 

•  The  first  phase  performed  a  critical  path  analysis  on  the  Pre-MS  B  activities  to 
identify  a  bottleneck  (see  the  Identification  of  Bottleneck  Activities  subsection). 

•  The  second  phase  focused  on  investigating  the  effect  of  reducing  the  process 
times  of  the  identified  bottleneck  activities  from  Phase  1  and  determining  the 
effect  of  allowing  them  to  be  executed  earlier  in  the  process  (see  the  Design  of 
Pre-MS  B  Bottleneck  Interventions  subsection). 

Identification  of  Bottleneck  Activities 

In  order  to  perform  critical  path  analysis,  the  Pre-MS  B  section  was  mapped  by  hand 
to  assist  in  visualization  of  the  complex  network  of  separation  and  batches  in  the  acquisition 
system.  The  processes  between  each  Separate  and  Batch  method  were  left  out  for 
simplicity  and  ease  of  interpretation.  The  section  or  line  segment  between  any  two  nodes 
was  labeled.  Figure  1  shows  the  mapped  version  of  the  Pre-MS  B  activities  and  Table  2 
shows  the  activities  associated  with  each  section. 
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Figure  1.  Pre-MS  B  Flowchart 


Table  2.  List  of  Activities  in  the  Pre-MS  B  Flowchart 

Section  Description  of  Activities 

1  Requirements  generation:  KPP  Development,  high  performance  team  work, 
etc. 

2  RFP  release,  contract  awarding 

3  Waiting  period  for  start  of  contract 

4-6  Cost  estimates  (contractor,  program  office,  and  independent) 

7  Affordability  assessment 

8  Set  acquisition  program  baseline 

9-10  No  processes 

1 1  Prepare  and  conduct  acquisition  panels 

12  Early  Systems  Engineering  (SE)  activities:  EOA,  developmental  testing, 
SRR,  etc. 

13  Acquisition  planning  activities 

14  Draft  RFP 

15  RFP  coordination 

16  Source  selection  plans 


Several  Assign  and  Record  modules  were  added  to  the  Arena  model  in  order  to 
determine  the  time  to  complete  each  segment.  Next,  a  trial  of  3,000  runs  was  performed  in 
Arena,  and  the  times  for  each  segment  were  collected.  A  spreadsheet  was  then  used  to 
analyze  results  and  to  determine  the  time  of  every  possible  path  from  the  beginning  of  the 
Pre-MS  B  activities  to  the  MS  B  decision.  The  path  that  took  the  longest  amount  of  time  was 
deemed  as  the  critical  path.  By  comparing  segments  in  the  longest  paths  to  the  sections 
found  in  shorter  paths,  the  bottleneck  activities  for  the  Pre-MS  B  processes  were  identified. 
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Design  of  Pre-MS  B  Bottleneck  Interventions 

In  order  to  improve  the  performance  and  alleviate  the  delay  caused  by  this 
bottleneck,  two  intervention  strategies  were  developed  and  tested  in  ERAM.  The  first 
intervention  performed  was  to  test  the  effect  of  decreasing  the  process  time  for  all 
bottleneck  activities.  In  order  to  test  the  effect  of  reducing  total  process  time,  the  minimum, 
maximum,  and  mode  for  these  activities  was  reduced  by  a  fixed  percentage.  A  paired  t-test 
was  then  performed  to  compare  each  trial  to  the  baseline  at  95%  confidence  level.  The 
reduction  by  using  a  fixed  percentage  was  performed  until  a  statistically  significant  change 
was  obtained.  Furthermore,  the  second  intervention  was  a  sensitivity  analysis  to  determine 
the  effect  of  allowing  the  bottleneck  to  be  performed  earlier  in  the  Pre-MS  B  process  to 
determine  its  effect  on  the  total  process  time.  The  results  of  these  interventions  are 
illustrated  in  the  next  section. 

Results  and  Discussions 

This  section  presents  the  results  of  both  simulation  analysis  phases  performed  on 
the  ERAM  Arena  model.  Specifically,  the  Pre-MS  B  Critical  Path  Analysis  Results 
subsection  presents  the  results  of  the  identification  of  the  critical  path  and  bottleneck 
activities.  Additionally,  the  Additional  Pre-MS  B  Bottleneck  Interventions  subsection  shows 
the  results  of  the  interventions  performed  on  the  bottleneck  analysis  to  improve  program 
completion  time. 

Pre-MS  B  Critical  Path  Analysis  Results 

During  the  critical  path  analysis,  times  for  all  1 1  paths  through  the  system  were 
calculated.  The  paths  were  labeled  by  letters.  Each  path  was  composed  of  segments.  A 
subset  of  the  paths  and  their  corresponding  activities  is  shown  in  Table  3. 


Table  3.  List  of  Paths  and  Segments  for  Pre-MS  B 


Path  Name 

Corresponding  Segments  From  Figure  1 

A 

1 

B 

2,  12,  14,  16,  10,  11 

C 

2,  12,  14,  15,  10,  11 

D 

2,  3,  6,  7,  8,  11 

E 

2,  3,  5,  7,  8,  11 

F 

2,  3,  4,  7,  8,  11 

G 

2,  3,  6,  7,  9,  10,  11 

H 

2,  3,  5,  7,  9,  10,  11 

1 

2,  3,  4,  7,  9,  10,  11 

J 

2,  3,  13,  14,  15,  10,  11 

K 

2,  3,  13,  14,  16,  10,  11 

As  seen  from  the  Table  3,  paths  B  and  C  heavily  overlap  while  path  A  has  no  overlap 
with  any  other  path.  From  the  total  time  for  each  path,  the  longest  was  deemed  the  critical 
path.  The  second  longest  and  third  longest  paths  were  also  determined.  A  subset  of  this 
data  can  be  seen  in  Table  4. 
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Table  4.  Length  of  Longest  Paths  to  MS  B 


Run 

Percentage 
of  Runs  as 
Longest  Path 

Percentage  of 
Runs  as 
Second 
Longest  Path 

Percentage  of 
Runs  as  Third 
Longest  Path 

Percentage  of  Runs 
in  the  Top  Three 
Longest  Path 

A 

45 

8.75 

42.5 

96.25 

B 

43.75 

38.75 

7.5 

90.00 

C 

6.25 

38.75 

31.25 

76.25 

D 

3.75 

8.75 

8.75 

21.25 

F 

0 

2.5 

5 

7.5 

J 

0 

1.25 

1.25 

2.50 

K 

1.25 

1.25 

3.75 

6.25 

E,  G,  H, 1 

0 

0 

0 

0 

As  can  be  observed  in  Table  4,  the  critical  path  was  most  often  A,  B,  and  C.  In 
approximately  95%  of  the  trials,  either  A,  B,  or  C  composed  the  critical  path.  Specifically, 
50%  of  the  time  path  B  or  C  was  the  critical  path,  and  45%  of  the  time  path  A  was  the  critical 
path.  We  note  that  path  B  and  C  have  significant  overlap;  therefore,  they  are  considered  as 
a  single  path,  path  B/C.  Since  the  critical  path  was  very  evenly  split  between  path  A  and 
path  B/C,  it  can  be  deduced  that  a  Pre-MS  B  process  common  to  both  of  paths  would  be  the 
bottleneck  of  the  process. 

In  examining  the  ERAM,  it  can  be  gleaned  that  there  was  some  interaction  between 
path  A  and  path  B/C.  One  of  the  last  modules  of  path  A  was  a  hold  module  called  “Wait  for 
EOA  completion.”  A  screenshot  of  this  module  can  be  seen  in  Figure  2. 


Figure  2.  Wait  for  EOA  Completion  Screenshot 

As  seen  in  Figure  2,  path  A  must  wait  for  the  EOA  to  be  complete  before  the  path 
can  finish.  A  second  communication  occurs  between  the  two  paths.  In  order  for  the  SE 
activities,  like  the  EOA,  to  occur,  the  Key  Performance  Parameters  (KPPs)  must  be 
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complete.  The  hold  model  called  “Wait  for  T&E  start”  facilitates  this  communication  and  can 
be  seen  in  Figure  3. 


Figure  3.  Wait  for  T&E  start  Screenshot 

However,  we  note  that  this  hold  module  also  waits  for  the  75%  of  the  contract  length 
to  elapse.  At  the  default  settings,  the  KPPs  will  always  be  completed  in  less  than  75%  of  the 
contract  length.  Therefore,  at  the  default  settings,  this  hold  does  not  serve  as 
communication  between  the  paths. 


Since  the  completion  of  the  EOA  was  the  only  communication  between  the  two 
critical  paths,  the  SE  activities  that  begin  before  the  EOA  completion  was  determined  to  be 
the  bottleneck  of  the  Pre-MS  B  activities.  If  this  bottleneck  activity  were  removed,  the  time  to 
MS  B  would  be  reduced  by  an  average  of  6.8%. 


Additional  Pre-MS  B  Bottleneck  Interventions 

Table  5  summarizes  the  results  of  the  t-tests  performed  for  when  the  process  time 
for  MS  B  system  engineering  activities  was  reduced.  The  tables  show  a  subset  of  trials 
corresponding  to  a  reduction  in  process  times  by  0%,  20%,  35%,  or  50%.  These  settings 
were  selected  to  show  the  sensitivity  of  the  model  to  various  degrees  of  process  time 
reduction.  From  these  simulation  analyses,  the  mean  ith% )  and  standard  deviation  of  the 

total  completion  time  for  each  trial  were  calculated.  These  calculated  means  were  compared 
to  the  mean  of  the  baseline  setting  (iibaSe)  in  the  default  settings,  or  0%  process  time 
reduction.  The  null  hypothesis  for  the  t-tests  is  H0-.  nbase  =  ii jth%  which  corresponds  to  a 
failure  to  reject  the  claim  that  the  baseline  and  the  ith  percentage  are  similar  and  alternative 
hypothesis  Ht\  iibase  A  nith0/o  if  there  is  significant  difference. 
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Table  5.  Summary  of  t-Test  Results  of  Process  Time  Reduction  for  SE  Activities 


%  Reduction  of  Process  Time 

0% 

(Baseline) 

20% 

35% 

50% 

Average  Time  to  MS  B 
(Days) 

3418.01 

3274.90 

3211.564 

3164.25 

Standard  Deviation  (Days) 

1701.08 

1636.108 

1557.816 

1515.48 

P-Value 

0.281 

0.109 

0.046 

Conclusion 

Fail  to  Reject 

Fail  to  Reject 

Reject 

Ho 

Ho 

Ho 

As  seen  in  Table  5,  it  is  evident  that  when  the  process  time  for  SE  activities  was 
decreased  by  less  than  50%,  there  will  not  be  a  statistically  significant  decrease  in  the  time 
to  MS  B.  However,  when  the  process  times  for  SE  activities  are  reduced  by  more  than 
approximately  50%,  the  model  exhibits  a  statistically  significant  decrease  in  time  to  MS  B. 

Based  on  the  identified  bottleneck,  which  was  the  SE  activities,  a  second  intervention 
was  developed.  Specifically  a  sensitivity  analysis  was  done  to  test  the  effect  of  allowing  the 
bottleneck  activities  to  occur  earlier  in  the  contract.  This  was  implemented  by  adjusting  the 
module  called  “Begin  Testing  PreB,”  which  can  be  seen  in  Figure  4. 


Figure  4.  Begin  Testing  PreB  Screenshot 

The  “Begin  Testing  PreB”  module  is  a  Decide  module  that,  when  set  to  true,  triggers 
the  beginning  of  the  SE  activities.  The  original  criteria  for  the  decide  module,  as  verified  and 
validated  by  Wirthlin  when  creating  the  ERAM  model,  was  that  75%  of  the  contract  length 
must  pass  before  these  activities  can  occur.  During  this  research,  this  percent  was 
decreased  to  simulate  the  SE  activities  occurring  sooner  and  more  resources  being  applied 
at  the  beginning  of  the  contract. 

In  addition,  to  allow  SE  tasks  to  begin  sooner,  the  KPPs  must  be  completed  sooner 
in  the  process  as  their  completion  is  also  needed  to  trigger  the  start  of  the  SE  tasks.  A  more 
complete  discussion  of  this  interaction  can  be  found  in  the  Pre-MS  B  Critical  Path  Analysis 
Results  subsection.  The  process  time  of  the  KPP  development  was  reduced  in  order  for  the 
KPPs  to  be  completed  in  a  manner  that  does  not  delay  the  SE  activities.  A  paired  t-test  was 
then  performed  to  compare  each  trial  to  the  baseline  at  95%  confidence  level. 
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Tables  6  and  Table  7  summarize  the  results  of  the  t-tests  performed  for  allowing  Pre- 
MS  B  contractor  activities  to  occur  earlier  in  the  contract.  Specifically,  Table  6  shows  the 
effect  of  allowing  the  SE  activities  to  occur  earlier  in  the  contract  when  the  KPPs  generation 
process  time  was  not  decreased,  and  Table  7  shows  the  effect  of  allowing  the  SE  activities 
to  occur  earlier  in  the  contract  in  conjunction  with  the  KPPs  generation  process  performing 
faster.  Table  6  shows  a  subset  of  trials  corresponding  to  the  SE  activity  starting  when  75%, 
50%,  33%,  or  25%  of  the  contract  has  elapsed.  Table  7  shows  a  subset  of  trials 
corresponding  to  the  SE  activity  starting  when  75%,  65%,  60%,  or  55%  of  the  contract  has 
elapsed. 

These  settings  were  selected  to  show  the  sensitivity  of  the  model  to  various  start 
times  of  SE  activities.  From  these  simulations,  the  mean  (nith %)  and  standard  deviation  of 
the  total  MS  B  completion  time  for  each  trial  were  calculated.  These  calculated  means  were 
compared  to  the  mean  of  the  baseline  setting  ( nbase )  'n  the  default  settings,  or  starting  after 
75%  of  the  contract  has  elapsed.  The  null  hypothesis  for  the  t-tests  is  H0\ iibase  =  [iith0/o 
which  corresponds  to  a  failure  to  reject  the  claim  that  the  baseline  and  the  ith  percentage  of 
contract  elapsing  before  start  is  similar  in  terms  of  program  completion  time  and  alternative 
hypothesis  //iibase  *  ^  there  is  significant  difference. 


Table  6.  Summary  of  t-Test  Results  of  SE  Activity  Start  Time  Adjustments  With 

Original  KPP’s  Process  Time 


%  of  Contract  Elapsed  Before  Start 

75% 

(Baseline) 

50% 

33% 

25% 

Average  Time  to  MS  B 
(Days) 

3418.01 

3379.09 

3379.09 

3379.09 

Standard  Deviation  (Days) 

1701.08 

1670.31 

1670.31 

1670.31 

P-Value 

0.770 

0.770 

0.770 

Conclusion 

Fail  to  Reject 

Fail  to  Reject 

Fail  to  Reject 

Ho 

Ho 

Ho 

Table  7.  Summary  of  t-Test  Results  of  SE  Activity  Start  Time  Adjustments  With 

Reduced  KPP’s  Process  Time 


%  of  Contract  Elapsed  Before  Start 

75% 

(Baseline) 

65% 

60% 

55% 

Average  Time  to  MS  B 
(Days) 

3418.01 

3305.44 

3200.75 

3139.95 

Standard  Deviation  (Days) 

1701.08 

1628.08 

1599.04 

1553.38 

P-Value 

0.392 

0.099 

0.032 

Conclusion 

Fail  to  Reject 

Fail  to  Reject 

Reject 

Ho 

Ho 

Ho 
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As  seen  in  Table  6,  it  is  evident  that  the  time  to  MS  B  is  not  sensitive  to  an  earlier 
start  time  for  SE  activities  when  the  KPP  process  time  is  set  to  the  default  distribution.  In 
fact,  when  the  start  time  is  at  50%,  33%,  and  25%  of  the  contract  time,  the  time  to  MS  B, 
standard  deviation  of  time  to  MS  B,  and  p-value  are  identical.  This  is  due  to  the  hold  module 
in  the  SE  path  described  earlier.  As  previously  discussed,  in  order  for  the  SE  activities  to 
begin,  a  percent  of  the  contract  must  elapse  and  the  KPPs  must  be  complete.  Once  the  SE 
activities  start  time  occurs  earlier  than  50%  of  the  contract  length,  the  KPPs  completion  is 
the  determining  factor  of  the  SE  activity  start  time. 

Table  7  takes  this  into  account  by  reducing  the  KPPs’  process  time  to  a  point  where 
it  does  not  dictate  the  start  of  the  SE  activities.  As  seen  in  Table  7,  it  is  evident  that  when  SE 
activities  begin  at  60%  of  the  contract  length  or  later,  there  will  not  be  a  statistically 
significant  decrease  in  the  time  to  MS  B.  However,  when  SE  activities  begin  at  55%  of  the 
contract  length  or  sooner  and  the  KPPs’  generation  processes  are  shortened  to  the  same 
degree,  the  model  exhibits  a  statistically  significant  decrease  in  time  to  MS  B. 

Conclusions 

The  critical  path  analysis  performed  in  this  research  indicated  that  the  SE  activities 
and  their  communication  with  the  requirements  branch  are  the  bottleneck  of  the  Pre-MS  B 
portion  of  the  acquisition  system.  In  addition,  the  research  indicated  that  focusing  on  reforms 
that  address  this  bottleneck  has  the  potential  to  decrease  the  total  time  spent  on  MS  B 
activities  by  approximately  7%;  this  corresponds  to  a  process  time  reduction  of 
approximately  six  months. 

This  research  also  tested  two  strategies  to  address  this  bottleneck.  The  first  was 
reducing  the  process  time  of  all  SE  activities.  The  second  was  to  allow  the  SE  activities  to 
have  an  earlier  start  time.  This  research  showed  that  the  latter  policy  has  the  potential  to  be 
the  most  beneficial.  This  research  showed  that  the  process  times  for  all  SE  activities  must 
be  decreased  by  approximately  50%  in  order  for  a  statistically  significant  decrease  in  time  to 
MS  B  to  occur.  This  degree  of  process  time  reduction  may  be  infeasible.  On  the  other  hand, 
allowing  the  SE  activities  to  occur  after  55%  of  the  contract  time  has  elapsed  rather  than  the 
current  75%  produces  a  statistically  significant  decrease  in  time  to  MS  B. 

The  increased  sensitivity  of  program  time  to  start  time,  rather  than  process  length, 
suggests  that  schedule  benefits  may  be  achieved  if  the  some  resources,  both  financial  and 
human,  are  transferred  from  the  SE  activities  to  the  activities  prior  to  test  and  development. 
However,  this  re-allocation  of  resources  must  be  accompanied  by  responsiveness  from  the 
JCIDS  branch,  which  is  the  branch  that  generates  the  KPPs.  This  research  indicates  that 
there  was  a  large  amount  of  co-dependence  between  the  JCIDS  and  SE  activities  and  that 
communication  and  coordination  between  these  branches  is  needed  in  order  to  address  the 
bottleneck. 
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